Repository navigation
[PowerX] use the configurable AMD exporter and native power telemetry - #3781
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @edwingao28's task in 1m 49s —— View job PR #3781 review: one blocking changelog issue
🔴 Blocking: The launch and power code had no blocking issues:
One non-blocking note: the multinode eval upload no longer includes |
|
|
||
| - config-keys: | ||
| - "*" | ||
| description: |
There was a problem hiding this comment.
one perf changelog entry as discussed earlier
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
moving srt slurm submodule bump to separate pr and rebasing this one |
4a64d00 to
4172b89
Compare
4a64d00 to
4172b89
Compare
cquil11
changed the title
There was a problem hiding this comment.
Beyond the inline finding, I also checked the dropped matrix.config['require-power'] fallback for test-sweep-single-node (e2e-tests.yml:509): single-node fixed-sequence recipes are validated to reject a per-recipe require-power field (infx/matrix/generate.py rejects it outside the one scoped multinode fixed-sequence), so only the global inputs.require-power flag can ever apply there — the missing fallback is not a behavior change.
Extended reasoning...
The PR re-pins srt-slurm to an upstream producer, reworks single-node power staging/validation in collect.py, and touches several CI workflow wiring points for require-power propagation; no auth/crypto surface is involved. A confirmed finding (collect.py:76) about a new recursive copytree that can turn log-staging I/O errors into benchmark-run failures is being posted inline, so approval is not warranted regardless. I independently verified one of the ruled-out candidates (the single-node require-power fallback removal) against infx/matrix/generate.py's validation, which rejects a per-recipe require-power flag for single-node fixed-sequence configs, confirming it is a non-issue.
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
a88e001 to
6dc48b1
Compare
中文：将主分支合并到 AMD 功耗遥测 PR。
The launcher no longer re-reads the bound recipe, patches its request, or writes $GITHUB_ENV. REQUIRE_POWER alone sets telemetry.required, so a single-node fragment may not set it, and result processing takes the producer SHA from the launcher stamp as the multi-node workflow does.
select_recipe already omits the native telemetry block for eval-only points, so the binder no longer rewrites telemetry.enabled.
Single-node and multi-node AgentX run the adapter through one helper, and the single-node adapter writes its sidecar into the workspace instead of the launcher copying it there.
The aggregate already records num_gpus from the same environment as GPU_COUNT, so the adapter drops --expected-num-gpus.
2 participants
Enable native srt-slurm GPU power collection for single-node fixed-sequence and AgentX jobs, including AMD's digest-pinned high-resolution exporter. Publish the native power package and result-side validation report, honor required-power failures, accept temperature samples, and reject packages whose producer recorded failed clock synchronization.
Rebased onto main after Alec's recipe-refactor stack merged. Native telemetry is composed through a shared fragment and bound after variant selection; exporter settings and workspace mounts come from cluster records. Recipe generation validates against each eligible cluster's exporter defaults. Eval-only jobs disable telemetry, and existing multi-node power policies remain in place.
Retains NVIDIA/srt-slurm#573 as a local patch to restrict worker-node samples to the job's participating GPUs. Includes the cleanup previously merged through #3798.
Replayed five saved hardware points through the refactored consumers: MI355X fixed-sequence and AgentX, plus B200 TP2/TP4 fixed-sequence and AgentX. All passed strict validation with every published power metric identical to the saved hardware results and original PR implementation.
Smoke runs on
d357d8b3b4:qwen3.5-fp8-mi355x-sglangqwen3.5-fp8-mi355x-sglang-agentic-mtpqwen3.5-fp4-b200-sglangminimaxm3-fp4-b200-trtllm-agentic-mtp